{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PDF \n",
    "> Portable Document Format (PDF), standardized as ISO 32000, is a file format developed by Adobe in 1992 to present documents, including text formatting and images, in a manner independent of application software, hardware, and operating systems.<br>\n",
    "可移植文档格式 （PDF），标准化为 ISO 32000，是 Adobe 于 1992 年开发的一种文件格式，用于以独立于应用程序软件、硬件和操作系统的方式呈现文档，包括文本格式和图像。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content='LayoutParser : A Uniﬁed Toolkit for Deep\\nLearning Based Document Image Analysis\\nZejiang Shen1( \\x00), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\\nLee4, Jacob Carlson3, and Weining Li5\\n1Allen Institute for AI\\nshannons@allenai.org\\n2Brown University\\nruochen zhang@brown.edu\\n3Harvard University\\n{melissadell,jacob carlson }@fas.harvard.edu\\n4University of Washington\\nbcgl@cs.washington.edu\\n5University of Waterloo\\nw422li@uwaterloo.ca\\nAbstract. Recent advances in document image analysis (DIA) have been\\nprimarily driven by the application of neural networks. Ideally, research\\noutcomes could be easily deployed in production and extended for further\\ninvestigation. However, various factors like loosely organized codebases\\nand sophisticated model conﬁgurations complicate the easy reuse of im-\\nportant innovations by a wide audience. Though there have been on-going\\neﬀorts to improve reusability and simplify deep learning (DL) model\\ndevelopment in disciplines like natural language processing and computer\\nvision, none of them are optimized for challenges in the domain of DIA.\\nThis represents a major gap in the existing toolkit, as DIA is central to\\nacademic research across a wide range of disciplines in the social sciences\\nand humanities. This paper introduces LayoutParser , an open-source\\nlibrary for streamlining the usage of DL in DIA research and applica-\\ntions. The core LayoutParser library comes with a set of simple and\\nintuitive interfaces for applying and customizing DL models for layout de-\\ntection, character recognition, and many other document processing tasks.\\nTo promote extensibility, LayoutParser also incorporates a community\\nplatform for sharing both pre-trained models and full document digiti-\\nzation pipelines. We demonstrate that LayoutParser is helpful for both\\nlightweight and large-scale digitization pipelines in real-word use cases.\\nThe library is publicly available at https://layout-parser.github.io .\\nKeywords: Document Image Analysis ·Deep Learning ·Layout Analysis\\n·Character Recognition ·Open Source library ·Toolkit.\\n1 Introduction\\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\\ndocument image analysis (DIA) tasks including document image classiﬁcation [ 11,arXiv:2103.15348v2  [cs.CV]  21 Jun 2021', metadata={'source': 'https://arxiv.org/pdf/2103.15348.pdf', 'page': 0})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pip install pypdf\n",
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "loader = PyPDFLoader(\"https://arxiv.org/pdf/2103.15348.pdf\")\n",
    "pages = loader.load_and_split()\n",
    "pages[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'LayoutParser : A Uniﬁed Toolkit for DL-Based DIA 5\\nTable 1: Current layout detection models in the LayoutParser model zoo\\nDataset Base Model1Large Model Notes\\nPubLayNet [38] F / M M Layouts of modern scientiﬁc documents\\nPRImA [3] M - Layouts of scanned modern magazines and scientiﬁc reports\\nNewspaper [17] F - Layouts of scanned US newspapers from the 20th century\\nTableBank [18] F F Table region on modern scientiﬁc and business document\\nHJDataset [31] F / M - Layouts of history Japanese documents\\n1For each dataset, we train several models of diﬀerent sizes for diﬀerent needs (the trade-oﬀ between accuracy\\nvs. computational cost). For “base model” and “large model”, we refer to using the ResNet 50 or ResNet 101\\nbackbones [ 13], respectively. One can train models of diﬀerent architectures, like Faster R-CNN [ 28] (F) and Mask\\nR-CNN [ 12] (M). For example, an F in the Large Model column indicates it has a Faster R-CNN model trained\\nusing the ResNet 101 backbone. The platform is maintained and a number of additions will be made to the model\\nzoo in coming months.\\nlayout data structures , which are optimized for eﬃciency and versatility. 3) When\\nnecessary, users can employ existing or customized OCR models via the uniﬁed\\nAPI provided in the OCR module . 4)LayoutParser comes with a set of utility\\nfunctions for the visualization and storage of the layout data. 5) LayoutParser\\nis also highly customizable, via its integration with functions for layout data\\nannotation and model training . We now provide detailed descriptions for each\\ncomponent.\\n3.1 Layout Detection Models\\nInLayoutParser , a layout model takes a document image as an input and\\ngenerates a list of rectangular boxes for the target content regions. Diﬀerent\\nfrom traditional methods, it relies on deep convolutional neural networks rather\\nthan manually curated rules to identify content regions. It is formulated as an\\nobject detection problem and state-of-the-art models like Faster R-CNN [ 28] and\\nMask R-CNN [ 12] are used. This yields prediction results of high accuracy and\\nmakes it possible to build a concise, generalized interface for layout detection.\\nLayoutParser , built upon Detectron2 [ 35], provides a minimal API that can\\nperform layout detection with only four lines of code in Python:\\n1import layoutparser as lp\\n2image = cv2. imread (\" image_file \") # load images\\n3model = lp. Detectron2LayoutModel (\\n4 \"lp :// PubLayNet / faster_rcnn_R_50_FPN_3x / config \")\\n5layout = model . detect ( image )\\nLayoutParser provides a wealth of pre-trained model weights using various\\ndatasets covering diﬀerent languages, time periods, and document types. Due to\\ndomain shift [ 7], the prediction performance can notably drop when models are ap-\\nplied to target samples that are signiﬁcantly diﬀerent from the training dataset. As\\ndocument structures and layouts vary greatly in diﬀerent domains, it is important\\nto select models trained on a dataset similar to the test samples. A semantic syntax\\nis used for initializing the model weights in LayoutParser , using both the dataset\\nname and model name lp://<dataset-name>/<model-architecture-name> .'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pip install rapidocr-onnxruntime\n",
    "loader = PyPDFLoader(\"https://arxiv.org/pdf/2103.15348.pdf\", extract_images=True)\n",
    "pages = loader.load()\n",
    "pages[4].page_content"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain0_1",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
